Explainable AI¶
Easy and hard way¶
In this project I demonstrate two ways to look inside a model and grasp what it looks at when classifying images into classes.
In the easy way, I use DataRobot to predict the type of food from images. DataRobot is an end-to-end enterprise AI platform that automates every step from data to value. It allows to train models for various tasks without writing a single line of code. You can try it for free for a certain amount of computations and after that you need to pay for using the tool.
In the more challenging (and interesting) way, I program a grad-CAM function in python to analyse the Brain tumor detector -model that I made in another project.
Grad-CAM visualisation - theory¶
Gradient-Weighted Class Activation Mapping (Grad-CAM) makes it possible to visualize the regions of the input that contributed towards making prediction by the model. It does so by using the class- specific gradient information flowing into the final convolutional layer of CNN to localize the important regions in the image that resulted in predicting that particular class.
Steps¶
To visualize the activation maps, first the image has to be passed through the model to make the prediction. Using argmax find the index corresponding to the maximum value in the prediction - this is the predicted class.
Grad-CAM is class specific, and you have to first choose the class $c$.
Then choose the layer, which feature maps will be analysed $$A^k \epsilon ℝ^{H \times W} $$ where $k=1,...,K$ indexes the channels.
Next, the gradient that is used to arrive to the predicted class from the feature map activations $A^k$ is calculated. $\alpha^c_k$ represents how important channel $k$ is for class $c$: $$ \alpha^c_k = \frac{1}{Z} \sum_i \sum_j \frac{\partial y^c}{\partial A^k_{ij}}$$ where $$\frac{1}{Z} \sum_i = \text{global average pooling}$$ $$\frac{\partial y^c}{\partial A^k_{ij}} = \text{gradients via backdrop}$$
To enhance the filter values that resulted in this prediction, the values are multiplied with tensorflow.GradientTape() with the filter values in the last convolutional layer.
This enhances the filter values that contributed towards making this prediction and lower the filter values that didn't contribute.
Then, the weighted combination of activation maps is performed and followed by ReLU to obtain the heatmap. $$ L^c_{Grad-CAM}= ReLU(\sum \alpha^c_kA^k) $$ where ReLU keeps only positive evidence, i.e. features that increase the class score, ignoring negative evidence.
If your goal is to visualise magnitude of influence/sensitivity, instead of support for class $c$, then replasing ReLU with $abs$ can also be helpful visualisation.
Finally, the feature heatmap is super-imposed on the original image to see the activation locations in the image.
Intuition¶
Grad-CAM highlights where the networks is "looking" by weighting conv features according to how much changing them would change the chosen class score. It is not a perfect explanation, just a sensitivity based localiation.What usually happens in the different layers of a image analysis network:
- early layers: edges, color contrasts, simple textures
- mid layers: motifs, parts, curves, repeated patterns
- deep layers: more class relevant patterns like parts of the object or relevant regions
Part 1. Easy way¶
In this section, I used DataRobot to build a model to classify images of food into categories. DataRobot has a built in tool to visualise the attention maps and heatmaps for Grad-CAM.
Food AI¶
The original dataset from https://www.kaggle.com/vermaavi/food11 consists of 16643 color images belonging to 11 categories. Due to data limitations in DataRobot, I will use pictures from 4 classes only:
- Dessert
- Seafood
- Fried food
- Vegetable-Fruit
The interface of DataRobot allows you to drop data in and then wrangle it. The system is built in such a way, that you don't need to know programming to train and rank models.
The best performing model for this task was Regularized Logistic Regression (L2).
| Metric | Validation | Cross-validation | Holdout |
|---|---|---|---|
| AUC | 0.9788 | 0.9877 | 0.9822 |
| Accuracy | 0.8900 | 0.9188 | 0.9036 |
| Balanced Accuracy | 0.8885 | 0.9196 | 0.9045 |
| FVE Multinomial | 0.7699 | 0.8202 | 0.7815 |
| LogLoss | 0.3188 | 0.2475 | 0.3019 |
Below is an example from the DataRobot models attention map for the final layer.
Part 2. Hard way¶
Grad-CAM for the Brain tumor detector classifier model¶
Next I will implement my own Grad-CAM pipeline in python. I wanted to see how the brain tumor detection happens. This model takes MRI images of the brain as input and classifies them into two classes: contains a brain tumor or not. You can check the project here: Brain tumor detector
Introduction to the Brain Tumor Detection¶
Deep learning has proven to be as good and even better than humans in detecting diseases from X-rays, MRI scans and CT scans. there is huge potential in using AI to speed up and improve the accuracy of diagnosis. This project will use the labeled dataset from https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation which consists of 3929 Brain MRI scans and the tumor location. The final pipeline has a two step process where
- A Resnet deep learning classifier model will classify the input images into two groups: tumor detected and tumor not detected.
- For the images, where tumor was detected, a second step is performed, where a ResUNet segmentation model detects the tumor location on the pixel level.
Below is an exmaple of an MRI image and the matching mask. This example has a small tumor. In images where no tumor is present, the mask will be complety black.
Convolutional neural networks (CNNs)¶
- The first CNN layers are used to extract high level general features
- The last couple of layers will perform classification
- Locla respective fields scan the image first searching for simple shapes such as edges and lines
- The edges are picked up by the subsequent layer to form more complex features
A good visualisation of the feature extraction with convolutions can be found at https://setosa.io/ev/image-kernels/
ResNet (Residual Network)¶
- As CNNs grow deeper, vanishing gradients negatively imapct the network performance. Vanishing gradient occurs when the gradient is backpropagated to earlier layers which results in a very small gradient.
- ResNets "skip connection" feature can allow training of 152 layers wihtout vanishing gradient problems
- ResNet adds "identity mapping on top of the CNN
- ResNet deep network is trained with ImageNet, which contains 11 million images and 11 000 categories
ResNet paper (He etal, 2015): https://arxiv.org/pdf/1512.03385
The ResNet architectures overcome the training challenges from deep networks compared ot the plain networks. ResNet-152 achieved 3.58% error rate on the ImageNet dataset. This is better than human performance.
Found 2839 validated image filenames belonging to 2 classes. Found 500 validated image filenames belonging to 2 classes. Found 590 validated image filenames belonging to 2 classes.
/usr/local/lib/python3.12/dist-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:121: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.
37/37 ━━━━━━━━━━━━━━━━━━━━ 183s 5s/step
Assessment of the model¶
The model accuracy is 0.94
Classification report
precision recall f1-score support
0 0.93 0.98 0.95 366
1 0.97 0.88 0.92 224
micro avg 0.94 0.94 0.94 590
macro avg 0.95 0.93 0.94 590
weighted avg 0.94 0.94 0.94 590
Grad-CAM for the Brain tumor detector model¶
Below is an example from the grad-CAM visualisation applied to the brain tumor image classifier model.
In the grid below we see that the attention map is corresponding well with the clinicians mask - eventhough the classifier has never seen or beentrained on the mask.
This block loops over a set of test images (here: cases predicted as containing tumor), loading each MRI slice and its ground-truth mask (if available), and then computing a Grad-CAM attention map for the classifier’s decision. The Grad-CAM attention map is contrast-enhanced for visibility. I chose to look further “in” than the very last convolutional layer (via the robust layer selection) because the final conv features can become too coarse or gradient-saturated, producing weak or even blank maps; earlier convolutional layers often preserve more spatial detail and yield attention maps that align better with the actual tumor region, making the explanation more informative and easier to interpret.
Below are examples of two layers
- conv5_block3_out, which is the last activaion layer in the network and
- conv5_block2_out, which is the activation layer after the second convolutional vlock. Sometimes the gradients can be very small so you can also see empty images. To see more examples from different images, click on the nodes in the model architecture.
[1/103] skipped input_layer: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor. [2/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv1_conv.png [3/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv1_relu.png [4/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_1_conv.png [5/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_1_relu.png [6/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_2_conv.png [7/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_2_relu.png [8/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_0_conv.png [9/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_3_conv.png [10/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block1_out.png [11/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_1_conv.png [12/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_1_relu.png [13/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_2_conv.png [14/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_2_relu.png [15/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_3_conv.png [16/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block2_out.png [17/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_1_conv.png [18/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_1_relu.png [19/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_2_conv.png [20/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_2_relu.png [21/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_3_conv.png [22/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv2_block3_out.png [23/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_1_conv.png [24/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_1_relu.png [25/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_2_conv.png [26/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_2_relu.png [27/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_0_conv.png [28/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_3_conv.png [29/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block1_out.png [30/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_1_conv.png [31/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_1_relu.png [32/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_2_conv.png [33/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_2_relu.png [34/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_3_conv.png [35/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block2_out.png [36/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_1_conv.png [37/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_1_relu.png [38/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_2_conv.png [39/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_2_relu.png [40/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_3_conv.png [41/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block3_out.png [42/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_1_conv.png [43/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_1_relu.png [44/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_2_conv.png [45/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_2_relu.png [46/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_3_conv.png [47/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv3_block4_out.png [48/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_1_conv.png [49/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_1_relu.png [50/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_2_conv.png [51/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_2_relu.png [52/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_0_conv.png [53/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_3_conv.png [54/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block1_out.png [55/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_1_conv.png [56/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_1_relu.png [57/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_2_conv.png [58/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_2_relu.png [59/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_3_conv.png [60/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block2_out.png [61/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_1_conv.png [62/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_1_relu.png [63/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_2_conv.png [64/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_2_relu.png [65/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_3_conv.png [66/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block3_out.png [67/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_1_conv.png [68/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_1_relu.png [69/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_2_conv.png [70/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_2_relu.png [71/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_3_conv.png [72/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block4_out.png [73/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_1_conv.png [74/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_1_relu.png [75/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_2_conv.png [76/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_2_relu.png [77/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_3_conv.png [78/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block5_out.png [79/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_1_conv.png [80/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_1_relu.png [81/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_2_conv.png [82/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_2_relu.png [83/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_3_conv.png [84/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv4_block6_out.png [85/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_1_conv.png [86/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_1_relu.png [87/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_2_conv.png [88/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_2_relu.png [89/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_0_conv.png [90/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_3_conv.png [91/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block1_out.png [92/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_1_conv.png [93/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_1_relu.png [94/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_2_conv.png [95/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_2_relu.png [96/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_3_conv.png [97/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block2_out.png [98/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_1_conv.png [99/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_1_relu.png [100/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_2_conv.png [101/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_2_relu.png [102/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_3_conv.png [103/103] saved -> /content/drive/MyDrive/Colab Notebooks/Explainable-AI/docs/pics/gradcam_by_layer/conv5_block3_out.png
Representation of different layers and their attention maps¶
Below is the architecture of the brain tumor detector. By clicking the convolutional layers and activation layers you can see examples of grad-CAM attention maps/heat maps side to side with the actual tumor mask from physicians.
Output hidden; open in https://colab.research.google.com to view.
Output hidden; open in https://colab.research.google.com to view.